Methodological Advisory Committee 
June 2002 


Using a Generalised Estimation Methodology 
for ABS Business Surveys 


Authors: John Preston and James Chipperfield 

c/ABS PO Box 10, Belconnen, ACT 2616 

fax: (02) 6252 8015 

email: john.preston@abs.gov.au phone: (02) 6252 6970 
email: james.chipperfield@abs.gov.au phone: (02) 6252 7301 


Part 1: Executive Summary 


1. The availability of Business Activity Statement (BAS) data collected by the 
Australian Taxation Of fice (ATO) has provided the Australian Bureau of Statistics 
(ABS) with opportunities to improve the efficiency of sample designs and estimations 
for its business surveys. The ABS business surveys currently use two methods of 
estimation; number-raised estimation and ratio estimation. While ratio estimation 
allows the use of one auxiliary variable to improve the precision of the estimates, 
generalised regression (GREG) estimation allows the use of more than one auxiliary 
variable, and hence has the potential to be more efficient than number-raised and 
ratio estimation. 


2. In order to realise these efficiencies, it will be necessary to develop a 
methodology that will produce generalised regression estimates and measures of 
accuracy of these generalised regression estimates for a variety of statistics under 
various sample designs. The linearization (or Taylor expansion) method and resampling 
(or replication) methods, such as Jackknife, Balanced Repeated Replications (BRR) and 
the Bootstrap, are generally used for non-linear statistics, such as ratio or regression 
estimators. While the linearization method involves the derivation of separate 
standard error formulae for each non-linear statistic, the resampling methods, 
although computationally intensive, require a single standard error formulae for all 
statistics. Hence, it was recommended that the various resampling methods should be 
evaluated for the ABS business surveys using generalised regression estimation. 


3. The choice of an appropriate variance estimation methods for point-in-time and 
movement estimates will depend to some extent on the underlying sample design 
methodology. The Grouped Partially Balanced Repeated Replication (GPBRR), the 
Simple Random Sampling With Replacement (SRSWR) Bootstrap, and the Simple 
Random Sampling Without Replacement (SRSWOR) Bootstrap options were considered 
for point-in-time estimates, while the GPBRR, one component SRSWR Bootstrap and 
SRSWOR Bootstrap, and the three component SRSWR Bootstrap and SRSWOR 
Bootstrap options were considered for movement estimates. 


4. A simulation study was conducted in two stages. The first stage was to 
compare the various variance estimation methods for number-raised and ratio 
point-in-time and movement estimates, while the second stage was to evaluate the 
"best" variance estimation method for GREG point-in-time and movement estimates. 
The first stage of the simulation study found that the Grouped Partially Balanced 
Repeated Replication (GPBRR) was not suited to the business survey's situation, and 
that the "best" variance estimation method is the SRSWOR Bootstrap for 
point-in-time estimates and the one component SRSWOR Bootstrap for movement 
estimates. The second stage of the simulation study found that the SRSWOR 
Bootstrap was also appropriate for GREG point-in-time estimates. At the time of this 
paper, the findings of the evaluation of the "best" variance estimation method for 
GREG movement estimates was not available. 


D: The recommendation of this paper is that the computer system should use the 
Bootstrap (where the selection of replicate samples are by SRSWOR) to produce 
estimated variances of the generalised regression estimates. This method is 
acceptable when measured against the four characteristics required in the computing 
system and the underlying methodology mentioned at the start of the paper, with one 
exception. The exception is that it is not possible to conclude with sufficient power 
that the generalised regression point-in-time and movement variance estimators are 
unbiased. 


6. The two alternative variance estimation methods are the Jacknife and Balanced 
Repeated Replication. The Jacknife is unsuitable because it requires many more 
replicate weights (i.e. the number of replicate weights needs to be at least equal to the 
sample size in each calibration class) for storage and processing. The Balanced 
Repeated Replication is unsuitable because of its simulation properties, specifically the 
efficiency of the variance estimator as measured by the root mean squared error. 


7. The questions for MAC members in relation to the developing of a methodology 
and a computer system that will produce generalised regression estimates and 
measures of accuracy of these generalised regression estimates are: 


o Is there sufficient evidence to support the recommendation to adopt the 
Bootstrap (SRSWOR)? If not, what should be done to obtain this evidence? 


o Is there sufficient theoretical and empirical evidence in the literature to 
support the hypothesis that the Bootstrap variance estimator is unbiased, or 
should the simulation study be extended? 


0 What is the minimum acceptable level of simulation error? (i.e. how many 
replicate weights should be used?) 


o What should be done to determine this acceptable level? 


o What brief guidance can be provided on the other issues listed in the 
conclusion? 


Part 2: The GREG Estimator and Variance Estimators 
2.1 Introduction 


8. The availability of Business Activity Statement (BAS) data collected by the 
Australian Taxation Office (ATO) has provided the Australian Bureau of Statistics 
(ABS) with opportunities to improve the efficiency of sample designs and estimations 
for its business surveys. The ABS business surveys currently use two methods of 
estimation; number-raised estimation and ratio estimation. While ratio estimation 
allows the use of one auxiliary variable to improve the precision of the estimates, 
generalised regression (GREG) estimation allows the use of more than one auxiliary 
variable, and hence has the potential to be more efficient (i.e. reduce the current 
sample sizes for ABS business surveys with no reduction in the accuracy of the 
estimates) than number-raised and ratio estimation. 


9. In order to realise these efficiencies, it will be necessary to develop a 
methodology that will produce generalised regression estimates and measures of 
accuracy of these generalised regression estimates for a variety of statistics under 
various sample designs; including: 


° 


single phase point-in-time estimates of level 
o single phase movement estimates of level 

o single phase point-in-time estimates of rates 
o single phase movement estimates of rates 

0 two phase point-in-time estimates of level 

0 two phase movement estimates of level 

0 two phase point-in-time estimates of rates 


0 two phase movement estimates of rates 


10. It will also be necessary to develop a computer system that will produce 
generalised regression estimates and measures of accuracy of these generalised 
regression estimates. There are several characteristics that would be required in this 
computing system, and the underlying methodology. Firstly, the computer system 
needs to be generic. This means that the algorithm for variance estimation is the 
same irrespective of the corresponding statistics. This simplifies the maintenance and 
development of the computing system. Secondly, the computer system needs to 
minimise the computation costs. These include the storage costs (e.g. the replicate 
weights in the case of the replicate variance estimators) and the time the computing 
system takes to calculate the variance estimators. Thirdly, the methodology needs to 
have desirable theoretical properties, such as unbiasedness. Fourthly, the 
methodology to have good empirical properties. This includes information about the 
bias and variance of the method that can be measured in simulation studies. All of 
these four characteristics can not be considered without consideration of the typical 
ABS sample design methodology (e.g. studies in the literature that simulation 
properties of variance estimation methods depend heavily on the sample design 
methodology). 


11. This part of the paper describes the generalised regression estimator for 
single phase and two-phase sample designs, and the various variance estimation 
methods for point-in-time and movement estimates for single phase sample designs. At 
the time of this paper, the various variance estimation methods had not been finalised 
for point-in-time and movement estimates for two-phase sample designs. 


2.2 The Generalised Regression Estimator 


12. The generalised regression estimator has several advantages over the standard 
Horvitz-Thompson estimator. Firstly, the estimates will be consistent with known 
population totals, and secondly, the estimates will usually be more accurate (for a fixed 
sample size). Furthermore, the generalised regression estimator is unbiased with 
respect to the assumed model, and is design consistent. 


2.2.1 Generalised Regression Estimator for Single-Phase Sample Designs 


13. Consider a finite population Cea tals , from which a probability sample 
s{scl) is drawn according to a sample design with selection probabilities 


A= 5), The sampling weights Ww, = Ya, are those used in the Horvitz- Thompson 
- = Vy; 
estimator, ies _ for variable of interest y. The objective is to estimate the 
= yi 
population total iv | where ” is the value of the variable of interest y for unit i. 


3 
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Assume there exists a set of auxiliary variables ~’ (Xie Mere Xe) for which the 

population totals U are Known. The generalised regression estimator is given 


by: 


-t 
B= bs ws | b W5 = i [> W5 


where 


14. The generalised regression estimator is often written as: 


bs, = DwG.Y; 


les 


where 7 the g-weight for unit i, defined as: 


les 


; 
Gg; -[tegr [4 -Zw | 


15. _—— Alternatively, the auxiliary information can be used to produce an unbiased 


a eat = MY; me | for 
estimator, es , by modifying the weights by a process known as calibration 


(Deville and Sarndal 1992). A new set of calibrated weights, “ , are sought which lie 
as close as possible to the set of design weights, “. The calibration requires the 
specification of distance function between the calibration weight and the design 
weight. Although any one of a number of distance functions could be used, the most 
commonly used is the generalised least squares distance function: 


p-y =m: 


les 


Minimisation of the generalised least squares distance function using Lagrange 

DW% =f, 
multipliers, subject to satisfying the calibration constraints, ‘<< , leads to the 
calibrated weights: 


where ies is the Horvitz-Thompson estimator for the set of auxiliary 
information. 


16. The calibrated estimator is equivalent to the generalised regression estimator: 


“an re E a 
bas AZ = 7, 1 +x/T* (4, = Fi] \y = DMG: = Treg 


les les les 


However, one disadvantage of the generalised regression estimator is that it can give 
negative weights, while under the calibrated estimator it is possible to impose range 


restrictions on the calibrated weights, L<g,sU _where 4 and Y are suitable lower 
and upper bounds. In order to satisfy the benchmark constraints and the range 
restrictions, the calculation of the calibration weights need to be undertaken using an 
iterative method (Singh and Mohl 1996). 


2.2.1 Generalised Regression Estimator for Two-Phase Sample Designs 


17. Consider a finite population Pe hela} _ from which a first-phase 


5.(5 SY) is drawn according to a sample design with selection 


S, (Ss cs5,cU) is 


probability sample 


probabilities 7 ~ Pree $1) A second-phase probability sample 


drawn from the first-phase sample according to a sample design with selection 


2 =Pr(/e s,|5,) 


probabilities - . The first-phase and second-phase sampling weights 


1 = YW; ang Me = Ve are those used in the two-phase estimator, 


ae =| DM iMei¥; = by WY; W. =WW 
rae es , for second-phase variable of interest y, where “/- "¥"2/, 


y=YVy. 
18. The objective is to estimate the population total 2 where ” is the 
value of the variable of interest y for unit i. Assume there exists two sets of auxiliary 
he = 2X 
. The population totals "0 are known for *, whereas the 
2; 


variables “1 and %2/ 


population totals are unknown for 
first-phase sample. 


, but the auxiliary variables are available for the 


19. The two sets of auxiliary information can be used to produce an approximately 
Deak = > WY, / 

unbiased estimator, ae , by modifying the weights using calibration. The 

first step is to minimise the generalised least squares distance function: 


2 


d= y hm) 


les, 1 


VW%,; = i 
subject to satisfying the first-phase calibration constraints, “4 . The 
minimisation of the generalised least squares distance function, using Lagrange 
multipliers, leads to the first-phase calibrated weights: 


a AF 
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ie = SMX 
where a is the first-phase estimator for the first set of auxiliary 
information. 


20. The second step is to minimise the generalised least squares distance function: 


ne “a 2 
pe > (W, — W, We; ) 
ESp Wi Wo, 
>. W, sf W,; ~t 
subject to satisfying the second-phase calibration constraints, ‘“* ee : 
where “/ ~ (Xi Xi) . The minimisation of the generalised least squares distance 


function, using Lagrange multipliers, leads to the second-phase calibrated weights: 


a 


a i 
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where ae is the first-phase estimator for the set of auxiliary information, 


and e% is second-phase estimator for the set of auxiliary information. 


21. Analternative approach (Hidiroglou, Estevao and Arcaro 2000) is to use the 


second-phase design weights “ ~ "2" as the starting weights in the second-phase 
calibration, rather than using “2 as the starting weights, and hence minimise the 
generalised least squares distance function: 


~ 2 
(W, — Wy; ) 
a= yl 
(eSp W, ;W2,; 
>, W, x; = W,; ~t 
subject to satisfying the second-phase calibration constraints, ‘“* ae ; 
where ~*~ (1. Xer) . The minimisation of the generalised least squares distance 


function, using Lagrange multipliers, leads to the second-phase calibrated weights: 


a 


Aa T 
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22. During the first-phase and second-phase calibration it is possible to impose 


range restrictions on the first-phase and second-phase calibrated weights, b= <4 


and LS Hi <U, where £ and Y are suitable lower and upper bounds. 


23. While these two alternative distance measures produce slightly different 
estimators, they have very similar properties. Since there was no obvious choice based 
on methodological reasons, the second method was chosen based on computer system 
efficiency reasons. 


2.3 Variance Estimators 


24. ‘The linearization (or Taylor expansion) method is often used for non-linear 
statistics, such as ratio or regression estimators, although resampling (or replication) 
methods, such as Jackknife, Balanced Repeated Replications (BRR) and the Bootstrap, 
are becoming more fashionable. While the linearization method is applicable to general 
sampling designs, the major disadvantage of this method is that it involves the 
derivation of separate standard error formulae for each non-linear statistic and hence 
additional computer programming. On the other hand, the resampling methods require 
a single standard error formulae for all statistics, but the major disadvantage of these 
methods is that they are computationally expensive. With the availability of more 
powerful computers, it is thought that the disadvantages associated with the 
resampling methods are minor compared to those associated with linearization 
methods. Hence, it was recommended that the various resampling methods should be 
evaluated for the ABS business surveys using generalised regression estimation. 


25. While the Jackknife is considered to be the most accurate of these resampling 
methods, it is also considered to be more costly and less timely in terms of processing 
than the other resampling methods (Shao and Tu 1995). Furthermore, the Jackknife 
requires many more replicate weights for storage and processing. Under the current 
estimation system for ABS business surveys, efficient computing procedures have 
been used to implement the Jackknife in order to substantially reduce these costs and 
improve the timeliness. Unfortunately, these procedures are only applicable for 
number-raised and ratio estimates and cannot be extended for use with generalised 
regression estimates. Hence, it was decided that the Jackknife would be excluded 
from the evaluation. 


26. The derivation of the point-in-time and movement variance estimators involved 
two steps. Firstly, the variance for the number raised estimator are explicitly derived. 
Secondly, an unbiased estimator of the number raised estimator with respect to the 
replicate sampling methodology are derived (e.g. SRSWR or SRSWOR). The derivation 
of the point-in-time and movement variance estimators extends to generalised 
regression estimation using a Taylor Series argument: the expectation of the replicate 
variance estimator, with respect to resampling, is equal to the Taylor Series variance 
estimator. (Note: the Taylor Series variance estimator is a linear combination of 
number raised variance estimates.) 


2.3.1 Variance Estimation Methods for Point-in-time Estimators 


27. The choice of an appropriate variance estimation method for point-in-time 
estimates will depend to some extent on the underlying sample design methodology. 
The majority of ABS business surveys are designed to produce reliable estimates fora 
number of variables across a number of classifications. The accuracy of the estimates 
for the classifications are generally controlled by stratifying the survey frame by 
these classifications (in the case where the classifications are available for all 
sampling units on the survey frame). The accuracy of the estimates are further 
improved by stratifying the sampling frame by other auxiliary variables related to the 
variables of interest. Therefore, the majority of ABS business surveys consist of a 
very large number of strata. 


Balanced Repeated Replications 


28. The standard procedure for the balanced repeated replications (BRR) is not well 
suited to the situation where there are a large number of strata, due to the cost of 
processing the large number of balanced replicates. In this situation it is possible to 
use a set of partially balanced replicates by dividing the large number of strata into a 
smaller number of groups (Wolter 1985). The procedure for the grouped partially 
balanced repeated replications (GPBRR) for a stratified random sample selected 
without replacement is to: 


(i) Randomly divide the original sample of units into two equal groups of 


m, =n, [2 units, independently within each stratum h. 
Pp ¥ 


(ii) Divide the total number of strata, H, into G groups with an equal number of 
strata within each group. 


(iii) Construct a set of R balanced half samples (replications) using Hadamard 
matrices, where R is a multiple of four greater than or equal to G. 


(iv) The GPBRR sampling weights within each balanced half sample r are given by: 


a an) fier 
(r) sf 


A 


Wr = 
w, (1 ee ifier 


(1, -m, ) 
where % ="%/Nb is the sampling fraction within stratum h. 
60,6 ae 
(v) Calculate the GPBRR estimators, ” ‘”’ ” , using these sampling weights: 
a,” = my; 


les 


(vi) The GPBRR variance estimator is given by: 


Bootstrap 


29. The standard bootstrap procedure for a stratified random sample selected 
without replacement (Shao and Tu 1995) is to: 


mM, 


(i) Select a simple random sample of “” units with replacement from the original 


sample of ” units, independently within each stratum h. Let % denote the 
number of times unit i in stratum h is included. The bootstrap sampling 
weights are given by: 


Ww, =wW . pee + eee ‘ “| 
, =W, (n, 1) (n,-1) ™m, hi 


=n,/N, 


where % is the sampling fraction within stratum h. 


Alternatively, if the simple random sample of ” units is selected without 


replacement from the original sample of ”» units, the bootstrap sampling 
weights are given by: 


The bootstrap estimator is calculated using these bootstrap sampling weights: 


6; = uy; 


les 


(ii) Independently replicate step (i) a large number of times, B, and calculate the 
A) a4) 
bootstrap estimates ” oily : 


(iii) The Monte Carlo approximation to the bootstrap variance estimator is given 


by: 
> (a ay 
0 -6,} 
Aa 7 y y 
Var (6, ) == Bal 
x Bon 
8, = 383" [a 
where b=1 


2.3.2 Variance Estimation Methods for Movement Estimators 


30. The choice of an appropriate variance estimation method for movement 
estimates will also depend to some extent on the sample design methodology used to 
produce the movement estimates. Most ABS repeated business surveys are designed 
to produce reliable point-in-time and movement estimates. The reliability of the 
point-in-time estimates are generally controlled through the allocation of the samples, 
while the reliability of the movement estimates are generally controlled through both 
the allocation and selection of the samples. Another objective of ABS repeated 
business surveys is to reduce provider load on individual businesses, and hence these 
surveys have also been designed to minimise the number of small and medium 
businesses (through sample allocation) as well as control the rotation of selected small 
and medium business (through sample selection). The sample selections for most ABS 
repeated business surveys are generally designed to enable a balance between reducing 
the compliance cost (i.e. provider load) on individual businesses and producing reliable 
movement estimates. The sample selections for many ABS repeated business surveys 
are performed using synchronised sampling (Brewer, Gross and Lee 2000). 


31.‘ The derivation of the various movement variance estimators under the sampling 
methodology used for ABS repeated business survey is best understood by means of a 
pictorial illustration (Figure 1). 


Figure 1: Pictorial Illustration of Sampling Methodology 
for ABS Repeated Business Surveys 
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32. The units selected in the survey can be partitioned into three categories: 
category (d) units are selected in the survey at time point 1 but not selected in the 
survey at time point 2; category (c) units are selected in the survey at time points 1 
and 2; and category (b) units are selected in the survey at time point 2 but not 
selected in the survey at time point 1. The derivation of variance estimation methods 


for movement estimates under this sample design methodology have not been 


considered in the literature. 


Balanced Repeated Replications 


33. 


The procedure for the grouped partially balanced repeated replications (GPBRR) 


for a stratified random sample selected without replacement is to: 


(i) Divide the units selected in the survey into the three categories (d,c,b) 
described above. 


(ii) Within each of the three categories (d,c,b) randomly divide the original 


“ch:"euh units into two equal groups of 


samples of “uh 
hay = Moan 21 Moy = Nop |@ 1 Moun = Moun [2 units, independently within each stratum 


h, where “tuh-“7cneur are the numbers of units selected in the survey in the 
three categories (d,c,b). 


(iii) Divide the total number of strata, H, into G groups with an equal number of 
strata within each group. 


(iv) Construct a set of R balanced half samples (replications) using Hadamard 
matrices, where R is a multiple of four greater than or equal to G. 


(v) The GPBRR sampling weights within each balanced half sample r are given by: 
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(vi) Calculate the GPBRR estimators, ”"’"”’ ”” , using these sampling weights: 
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Bootstrap 


34. The proposed bootstrap procedure for stratified random samples selected 
without replacement is to: 


(i) Divide the units selected in the survey can be partitioned into the three 
categories (d,c,b) described above. 


(ii) Within each of the three categories (d,c,b) select simple random samples of 


Munn Teuh units with replacement from the original samples of “hun Moh eur 


units, independently within each stratum h, where un Mchs"euh are the numbers 


of units selected in the survey in the three categories (d,c,b). 


Let i and 


denote the number of times unit i in stratum h is included at 


time points 1 and 2. The bootstrap sampling weights are given by: 
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Alternatively, if the simple random samples of “tu%n-eur units are selected 


without replacement from the original samples of "hun 


Nn Moun units, the 


bootstrap sampling weights are given by: 
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The bootstrap estimator is calculated using these bootstrap sampling weights: 


6, a > We:Yo; - » Ws; 


fe(d,c) fe(b,c) 


(iii) Independently replicate step (ii) a large number of times, B, and calculate the 
A) (8) 
bootstrap estimates ym 1-1 ym : 


(iv) The Monte Carlo approximation to the bootstrap variance estimator is given 


by: 
> (a 4 
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where b=1 


35.  Analternative to the bootstrap procedure described above is to split the 
variance estimator into three components: the variance at time 1; the variance at time 
2; and the covariance between time 1 and time 2. 


Var ( aye = Var (6,,)+Var (6,2) - 2x Cov (8 6 


ylevy2 


36. Thealternative bootstrap procedure for stratified random samples selected 
without replacement is to: 


(i) Divide the units selected in the survey into the three categories (d,c,b) 
described above. 


(ii) Within each of the three categories (d,c,b) select simple random samples of 
Mun McnTeuh units with replacement from the original samples of “tu ch eur 


units, independently within each stratum h, where Mun cheoh are the numbers 
of units selected in the survey in the three categories (d,c,b), where: 


Mun = tun * min{(1, rs 2)/M, (Moy - 2) (mi 
Mp = Nop min{(”, a 2)/n, Ane 2)/n.,} 
Mop = Noun * min{(1, 2)/n, Aiy= 2)/n,,} 


Let ‘i and 4 denote the number of times unit i in stratum h is included at 
time points 1 and 2. The bootstrap sampling weights are given by: 
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The bootstrap estimators are calculated using these bootstrap sampling 
weights: 
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iii) Independently replicate step (ii) a large number of times, B, and calculate the 
Pp y rep Pp g 
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(iv) The Monte Carlo approximation to the bootstrap variance estimator is given 
by: 


3.1 The Simulation Study 


37. The simulation study was conducted in two stages. The first stage was to 
compare the various variance estimation methods for number-raised and ratio 
point-in-time and movement estimates, while the second stage was to evaluate the 
"best" variance estimation method for GREG point-in-time and movement estimates. 
The first stage was conducted using number-raised and ratio estimates, rather than 
GREG estimates, since it was possible to compare the variances under the various 
estimation methods against the standard linearization variance using number-raised 
and ratio estimates. It was also assumed that the "best" variance estimation method 
for number-raised and ratio point-in-time and movement estimates would also be the 
"best" variance estimation method for GREG point-in-time and movement estimates. 


38. The simulation study was restricted to single phase point-in-time estimates of 
level and single phase movement estimates of level. The more complex point-in-time 
estimation statistics (e.g. single phase estimates of rates) will not be included in this 
evaluation, since it is considered that the "best" variance estimation method for single 
phase point-in-time level estimates will most likely be the "best" variance estimation 
method for the point-in-time estimates for the more complex point-in-time estimation 
statistics. 


39. The various variance estimation methods were assessed by comparing the 
relative root mean squared error percents. The relative root mean squared error 
percents were calculated as the square root of the estimated mean squared error (i.e. 
estimated bias squared plus estimated variance) of the various variance estimation 
methods divided by the standard linearization variance (expressed as a percentage). 
The estimated biases and estimated variances, and hence these estimated mean 
squared errors, were calculated using one hundred simulations of the variances under 
the various variance estimation methods: 


100, 
RMSE (Var (6, Is)} = > Geely 
x 


Var | 6, |s 

where | | is the standard linearization variance given the realised sample s and 

Var (6\"|s) | | | | _ 
is the estimated variance under the various variance estimation methods 

for simulation i given the realised sample s. 


40. At face value the comparison of the relative root mean squared error percents 
for the various variance estimation methods might appear to be quite unusual, since it 
is expected that: 


lim | RMSE (Var (8, s))| >0 


for all the various variance estimation methods, where R is the number of replicate 
samples. Therefore, it is expected that all the various variance estimation methods 
will be equivalent, under this relative root mean squared error criteria, for very large 
numbers of replicate samples. However, in practice the number of replicate samples 
will need to be kept to a minimum in order to keep the computer processing time and 
costs, as well as computer storage size and costs, to manageable levels. 


41. It should also be noted that the results of the simulation study compare the 
"resampling error" of the various variance estimation methods, and hence do not take 
into account "sampling error", since the results are based ona single original sample 
from selected from the population. 


3.2 The Results of the Simulation Study 


42. The first stage was to compare the various variance estimation methods for 
point-in-time and movement of number-raised and ratio estimates. The relative root 
mean squared error percents, under the various variance estimation methods, based on 
approximately 50 and 100 replicate samples, for the point-in-time number-raised 
estimates of sales from the "Quarterly Economic Activity Survey, March 2000" are 
presented in Table 1. 


43. While the estimated relative root mean squared error percents for the Grouped 
Partially Balanced Repeated Replications (GPBRR) were only slightly higher than those 
for the two Bootstrap options at the Australian level, they were considerably higher at 
the industry level. Furthermore, as the number of replicates increased from 48 to 96, 
there appeared to be little improvement in the estimated relative root mean squared 
error percents at the industry level for the GPBRR. These industry level results were 
not unexpected, since when the number of strata in an industry is less than 48 the 
GPBRR estimators with 48 and 96 replicates are equivalent. (The relative root mean 
squared error percents in Table 1 were not always exactly the same because the 
replicates groups were formed using different random starts). The GPBRR estimators 
with 48 and 96 replicates were not the same when the number of strata in an industry 
is greater than 48 (i.e. Industry 3 and 7). In these industries, it might be expected 
that the GPBRR estimator with 96 replicates will have much lower simulation error (as 
measured by relative root mean squared error percents) but this was not the case. It 
was decided that the GPBRR was probably not suited to the business survey's 
situation, and hence the GPBRR method was excluded from the remainder of the 
simulation study. 


Table 1: Relative Root Mean Squared Error Percents 
for Point-in-time Number-Raised Estimates of Sales, 
under Various Variance Estimation Methods, 
Quarterly Economic Activity Survey, March 2000 
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26.47 26.41 19.54 13.95 21.84 15.55 


2 
3 
4 
5 
7 
10 
11 
12 
14 
15 
16 


51.98 51.98 20.78 15.17 19.52 14.89 


29.76 29.21 20.10 15.40 18.74 12.46 


"The SRSWR Bootstrap was based on sample sizes of (m =n - 1), while the SRSWOR Bootstrap was based 
on sample sizes of (m= n/2). 


44. The estimated relative root mean squared error percents for the SRSWOR 
Bootstrap were consistently slightly lower than those for the SRSWR Bootstrap. An 
evaluation into an SRSWR Bootstrap based on larger sample sizes of (m= 5n) found 
that there was little improvement over the SRSWR Bootstrap based on sample sizes of 
(m =n - 1), and it was still not as good as the SRSWOR Bootstrap based on sample 
sizes of (m= n/2). 


45. The relative root mean squared error percents, under the various variance 
estimation methods, based on approximately 50 and 100 replicate samples, for the 
movement of number-raised estimates of sales from the "Quarterly Economic Activity 
Survey, March and June 2000" are presented in Table 2. 


46. The estimated relative root mean squared error percents for the three 
component SRSWR Bootstrap were consistently higher than the one component 
SRSWR Bootstrap and SRSWOR Bootstrap options. There was not much difference 
between the estimated relative root mean squared error percents for the SRSWR 
Bootstrap and SRSWOR Bootstrap options. 


Table 2: Relative Root Mean Squared Error Percents 
for Movement of Number-Raised Estimates of Sales, 
under Various Variance Estimation Methods, 
Quarterly Economic Activity Survey, March and June 2000 


13.34 20.23 19.20 19.42 
15.24 26.74 14.76 11.58 


38.86 
12.14 
10.71 


24.68 19.57 15.21 21.79 15.83 
29.72 29.48 24.41 21.66 19.75 


13.90 
44.37 


"The SRSWR Bootstrap was based on sample sizes of (m =n - 1), while the SRSWOR Bootstrap was based 
on sample sizes of (m= n/2). 


47. The various variance estimation methods were evaluated against other variables 
from the "Quarterly Economic Activity Survey", as well as another survey using 
number-raised and ratio estimates, with similar findings. Therefore, it is 
recommended that the "best" variance estimation method is the SRSWOR Bootstrap 
for point-in-time estimates and the one component SRSWOR Bootstrap for movement 
estimates. 


48. The second stage was to evaluate the "best" variance estimation method for 
point-in-time and movement of GREG estimates. The relative standard error percents, 
under the various variance estimation methods, based on approximately 50 and 100 
replicate samples, for the point-in-time GREG estimates of sales from the "Quarterly 
Economic Activity Survey, March 2000" are presented in Table 3. 


49. The estimated relative standard errors for the SRSWOR Bootstrap for the 
point-in-time GREG estimates were similar to those for the number-raised estimates. 
Therefore, it appears that the SRSWOR Bootstrap is also appropriate for 
point-in-time GREG estimates. At the time of this paper, the findings of the 
evaluation of the "best" variance estimation method for movement of GREG estimates 
was not available. 


Table 3: Relative Standard Error Percents 
for Point-in-time GREG Estimates of Sales, 
Quarterly Economic Activity Survey, March 2000 


Bootstrap SRSWOR 
Industry 


a 100 


4. Conclusion 


50. The recommendation of this paper is that the computer system should use the 
Bootstrap (where the selection of replicate samples are by SRSWOR) to produce 
estimated variances of the generalised regression estimates. This method is 
acceptable when measured against the four characteristics required in the computing 
system and the underlying methodology mentioned at the start of the paper, with one 
exception. The exception is that it is not possible to conclude with sufficient power 
that the generalised regression point-in-time and movement variance estimators are 
unbiased. 


51. —‘ The two alternative variance estimation methods are the Jacknife and Balanced 
Repeated Replication. The Jacknife is unsuitable because it requires many more 
replicate weights (i.e. the number of replicate weights needs to be at least equal to the 
sample size in each calibration class) for storage and processing. The Balanced 
Repeated Replication is unsuitable because of its simulation properties, specifically the 
efficiency of the variance estimator as measured by the root mean squared error. 


52. The variance estimation methods used in computer systems of other statistical 
agencies do not have acceptable characteristics. The methodology supporting the 
Statistics Canada package, GES, is Taylor Series which does not meet the generic 
characteristic. Also the methodology supporting the Statistics Netherlands package, 
BASCULA, is Balanced Repeated Replication, which has already been shown to be 
unsuitable. The documentation accompanying BASCULA proposes an adjustment to 
Balanced Repeated Replication to improve its efficiency. This adjustment is called 
Balanced Repeated Replication with pseudo strata. However, the cost of improving 
efficiency will result in the method being unsuitable due to an increase in the number 
of replicate weights. 


53. One variation of the Balanced Repeated Replication with pseudo strata is where 
the pseudo strata are partially, rather than completely balanced. The partial balancing 
arbitrarily reduces the number of replicate weights that are required by the complete 
balancing. However, the cost of partial balancing is a decrease in the efficiency of the 
variance estimator. Without a simulation study it is not possible to measure the 
efficiency of this variation to the Balanced Repeated Replication method. 


54. Apart from the issues raised in this paper, there are a number of other issues 
that needed to be considered in relation to developing a computer system that will 
produce generalised regression estimates and measures of accuracy of these 
generalised regression estimates. These issues include developing methods to deal 
with surprise outliers, non-response, undercoverage of survey frames, imputation, 
winsorisation and non-convergence of the calibrated replicate weights. 


5. Discussion Points for MAC Members 


55. The questions for MAC members in relation to the developing of a methodology 
and a computer system that will produce generalised regression estimates and 
measures of accuracy of these generalised regression estimates are: 


o Is there sufficient evidence to support the recommendation to adopt the 
Bootstrap (SRSWOR)? If not, what should be done to obtain this evidence? 


o Is there sufficient theoretical and empirical evidence in the literature to 
support the hypothesis that the Bootstrap variance estimator is unbiased, or 
should the simulation study be extended? 


o What is the minimum acceptable level of simulation error? (i.e. how many 
replicate weights should be used?) 


0 What should be done to determine this acceptable level? 


0 What brief guidance can be provided on the other issues listed in the 
conclusion? 
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